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Abstract 

This paper examines the performance trade-off when implementing a blockchain architecture for a 
cloud-based groupware communication application. We measure the additional cloud-based resources 
and performance costs of the overhead reąuired to implement a groupware collaboration system over a 
blockchain architecture. 

To evaluate our groupware application, we develop measuring instruments for testing scalability and 
performance of Computer Systems deployed as cloud computing applications. While some details of our 
groupware collaboration application have been published in earlier work, in this paper we reflect on a 
generalized measuring method for blockchain-enabled applications which may in turn lead to a generał 
methodology for testing cloud-based system performance and scalability using blockchain. 

Response time and transaction throughput metrics are collected for the blockchain implementation 
against the non-blockchain implementation and some conclusions are drawn about the additional 
resources that a blockchain architecture for a groupware collaboration application imposes. 

Keywords: blockchain, groupware, collaboration Systems, cloud computing, secure messaging 
Systems. 


265 





Australasian Conference on Information Systems 
2019 Perth Western Australia 


Beck, Eklund & Spasovski 
Blockchain groupware performance case study 


1 INTRODUCTION 

Group communication tools are increasingly used in work environments and can be characterized by 
many database changes that reąuire fast, secure, and scalable message Processing. These reąuirements 
can be a challenge to security and scalability. Are these aims simultaneously achievable in cloud-based 
groupware communication applications? If so, at what cost to system resources and performance? 

While group collaboration practices are intensifying in distributed teams, most collaboration tools such 
as Jive or Yammer or Slack 1 are architected around a trusted central server (Beck et al., 2014). In order 
to reflect the changing naturę of collaboration, decentralized collaboration tools are anticipated that are 
as secure as centralized Systems, but without the need for a trusted third-party (Ciriello et al., 2018). 
This will - among other features - allow users to control their data, even take their data with them when 
they move to the next assignment or job (Tapscott and Tapscott 2016). One natural way of achieving 
decentralization in group communication tools and permitting finer granular control over private data 
is to organize the groupware application around a peer-to-peer blockchain. 

While blockchain promises advantages over traditional databases, for instance strong irrefutability, 
auditable data storage (Zyskind et al., 2015; Nakamoto, 2008; Fisher and Sanchez, 2016) and without 
the need for a trusted third-party, however it also introduces complexity and implementation overhead. 
The strength of hardening data against tampering through the use of a decentralized ledger results in a 
performance and scalability challenge, mainly because of the overhead introduced by the consensus 
algorithms and the costs associated with distributing the ledger (Dinh et al., 2017). For this reason, it is 
important to better understand the advantages, disadvantages and costs of using blockchain in detail. 

In this paper we were interested in how blockchain-powered groupware collaboration 
applications scalę? As this reąuires cloud-based performance testing, we also answer the ąuestion 
how scalability and performance are benchmarked in blockchain-enabled groupware 
applications? The futurę objective of our work is to develop expertise that can predict performance 
impact and anticipate the resources needed to deploy blockchain-enabled applications. 

This paper is structured as follows. Section 2 presents some background on blockchain, describing the 
mechanisms used in its implementation and summarizing the performance characteristics of various 
blockchain implementation alternatives. Section 3 presents the design science research approach that 
was foliowed in our iterative development and testing cycle. Section 4 discusses the architecture used 
and gives details of the implementation of the collaboration application. In Section 5, we present the 
measuring instruments and the various test scenarios. In Section 6, the performance results are 
presented, focusing on the key metrics of response-time and throughput at varying levels of resource 
provisioning. In Section 7 we confront expectations with the experimental findings, thereafter we 
analyze each of the results. Lastly, in Section 8 we conclude with a brief summary of the results, their 
analysis and our futurę plan of work. 

2 BLOCKCHAIN BACKGROUND 

Consensus algorithms are used within blockchains to ensure that participating nodes in the distributed 
network agree with the State of the blockchain when new blocks are added (Christidis and Devetsikiotis, 
2016). Blockchain consensus algorithms have the property of “Byzantine fault tolerance”, meaning that 
no single machinę can succeed in a malicious attack on the distributed system (Lamport et al., 1982) 
without being ‘checked’ or ‘detected’ by other nodes in the distributed network. This is an important 
feature for a groupware communication tool because the architecture delivers strong irrefutability 
among the message group that, in practice, eliminates ex post facto amendments to the ledger. 

The first implementation of a blockchain in 2009 used the consensus algorithm ‘proof-of-work’ 
(Nakamoto, 2008). Since then, a variety of consensus algorithms have been developed and used, mostly 
variants of ‘proof-of-stake’ or ‘proof-of-authority’. 

Bitcoin is a cryptocurrency powered by a blockchain that allows users to submit transactions without 
the need for a centrally trusted organization; this is achieved using ‘proof-of-work’ consensus 
(Nakamoto, 2008). The proof-of-work algorithm is based on one-way hash functions, where the only 
mechanism to recover a hash key is via brute-force, namely without strategy or heuristic, to generate 
and test permutations of messages until they match the hash value. This undirected search approach 
reąuires a large amount of computational resource (and therefore energy), resulting in slow transaction 
times and smali throughput. 


1 https://www.iivesoftware.com. https://products. office.com/en-au/vammer/ . https://slack.com/ 
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‘Proof-of-stake’ is a less expensive consensus altemative to proof-of-work. Here, no brute-force 
computing is reąuired to achieve consensus. Instead, the next node to create a błock is selected 
proportional to its ‘stake’ in the blockchain. In cryptocurrency blockchains, the stake is the number of 
tokens a node holds - usually combined with how long they have been held (Bentov, et al., 2016). In non- 
cryptocurrency blockchains, stakes typically do not exist. Thus, an authority is selected to add blocks to 
the blockchain, usually via a ballot or lottery, referred to as ‘proof-of-authority’ consensus (Knirsch, et 
al., 2018). 


Framework Name 

Consensus Algorithm OpenSource 

Throughput (tx/s) 

Response time (secs) 

Bitcoin 

PoW 

Y 

3-5 

> 500 

Ethereum 

PoW 

Y 

15-30 

360 

Ethereum Casper 

PBFT/PoW hybrid - ethash 

Y 

» 5000 

unknown 

Ripple RPCA (Ripple Protocol consensus Algorithm 

Y 

50,000 

4 

NEO 

Delegated-BFT 

Y 

10,000 

Ol 

1 

10 

0 

Hyperledger Fabric 

Kafka/Raft 

Y 

80,000 

< 1 

Hyperledger Sawtooth 

Proof of Elapsed Time (PoET) 

Y 

morę than 80,000 

< 1 

MultiChain 

PBFT + MultiChain 

Y 

1,000-1,500 

5-10 

Qourum 

Raft/IBFT 

Y 

835 

5 

Tendermint 

Tendermint BFT 

Y 

4,000 - 10,000 

< l 

Red Belly 

Democratic-BFT 

N 

660,000 

2-4 

Kadena 

Scalable PoW-BFT 

N 

8,000 

< 0.1 


Table 1. Some blockchain frameworks and their performance claims extendedfrom 
Buchman, 2016 and Yukolic, 2017. 


Ali blockchains fali into one of two categories, namely ‘public’ or ‘private’ (Peters and Panayi, 2016). 
Public-blockchains allow all users on the network to view the data stored, while private blockchains hide 
data stored on the blockchain between permissioned participants. Private blockchains can be either 
fully-private or consortium blockchains. Read/write permissions of a fully-private blockchain are 
controlled by a single organization, while consortium blockchains distribute read/write permissions 
across a permissioned consortium, which adds the extra security of decentralization. Thus, while the 
consensus mechanism for measuring blockchain scalability and performance is important, so too is the 
governance structure and decision whether to use a public permissioned/permission-less or private- 
consortium blockchain (Beck et al., 2018). 

3 DESIGN SCIENCE METHODOLOGY 

For the development and evaluation of our groupware collaboration tool we follow a Design Science 
Research (DSR) approach (Simon, 1996). Like other prior DSR information system artefact approaches, 
we design, implement, and evaluate a new IT artifact (March & Smith, 1995; Orlikowski and Iacono, 
2001). In this case a blockchain-powered groupware collaboration tool, while simultaneously developing 
a method to compare the performance and scalability overhead to guide futurę blockchain Systems 
design and evaluation (Gregor and Hevner, 2013; Gregor and Jones, 2007). 

We follow a DSR approach because we aim for a methodological development and evaluation of an IT 
artifact, at the same time developing a way to compare the performance and scalability of blockchain- 
enabled applications morę generally. We followed the guidelines for theory-generating DSR by Beck et 
al. (2013) including (1M4) as follows, these steps are iterative (Beck et al., 2013). 

a) Creating awareness of the problem and suggesting an approach to solve it: Poor scalability and 
performance are two routine pitfalls using blockchain Systems (Tschorsch and Scheuermann, 
2016), this is partly due to the restrictions of the core technology, but the design and architecture 
of the underlying application can also impact performance. Our approach is to design the 
architecture and functionality with an emphasis on minimizing vulnerabilities exposed by the use 
of the blockchain. 

b) Deyeloping the artefact: The application was developed iteratively, after each stage tested and 
evaluated and subseąuent improvements madę. The first iteration involved the architecture of the 
software, where the focus was on obtaining best performance and scalability by following common 
software patterns (Richards, 2015). The second iteration focused on creating the core 
functionality of the application using ‘Separation of Concerns’ (Hiirsch and Lopes, 1995). 
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c) Eyaluating the artefact: After an initial development stage the architecture was evaluated and a 
decision taken on whether an alternative architectural pattern would be better suited. This 
architectural pattern revision took advantage of non-validator blockchain nodes, i.e., not every 
client participates in consensus-making. This design modification improved performance and 
scalability. After the second stage we analysed the core functionality, and pin-pointed bottlenecks 
based of micro-benchmarks of the blockchain technology used (Buchman, 2016). The evaluation 
of the core functionality led to the decision that using non-blocking (asynchronous) functions 
could circumvent validator response time, thus improving performance. Using asynchronous 
functions is a design choice that was well suited to this use case, but does not suit all use cases. 

d) Abstracting design knowledge: In order to accurately abstract design knowledge and measure 
efficiency, we created a tool to measure scalability and performance based on a performance 
testing methodology (Menasce, 2002). The measurement of scalability and performance can be 
subjective, depending on the underlying cloud-based infrastructure, thus we reduce this idea to 
an objective comparison between two Systems: comparing an implementation without 
blockchain to a system with the same underlying infrastructure with a blockchain 
implementation. This allows us to measure the overhead of the blockchain relative to the base- 
line performance of the artifact with and without the features provided by the blockchain. In the 
following text we will explain in morę detail the developed collaboration tool and how the 
performance and scalability tests are applied. 

4 BLOCKCHAIN-BASED COLLABORATION TOOL IMPLEMENTATION 

The core functionality of the collaboration tool is powered by a RESTful API built in Scala which sends 
and receives messages between groupware clients, otherwise referred to as ‘the application’. The 
blockchain used in our experiment is the permissioned private consortium blockchain Tendermint 2 * 4 . 
Tendermint is a relatively lightweight blockchain solution. Its consensus method is ‘proof-of-authority’ 
using a voting mechanism (Buchman, 2016), as opposed to the morę computational expensive proof-of- 
work madę famous by Bitcoin. 

Two different node types exist in Tendermint, namely validator and non-validator nodes. Validator 
nodes are part of the consortium, nodes which vote to agree on consensus, while non-validator nodes 
are restricted to reading and proposing transactions on the blockchain. All nodes in the Tendermint 
blockchain communicate over a persistent encrypted TCP P2P gossip communication protocols. 

From a design perspective, a straight-forward method of using a blockchain is to storę all data in the 
blockchain. Storing all data has one significant drawback; namely if all data stored on the blockchain is 
immutable, no data can ever be removed. This in turn causes the blockchain to grow so large that it can 
become impractical to storę and distribute, particularly for a groupware messaging application. 

Growth of the blockchain is inevitable but the ratę of growth can be managed. All data in our 
implementation is stored in MongoDB4, while only a hash-key of the data entries is saved in the 
blockchain. The hash-keys in the blockchain are used as a check to confirm the validity of the data stored 
in the database. For each individual data-entry retrieved from the database, the blockchain is ąueried 
with the appropriate hash-keys, thus confirming the validity the data retrieved from the database. A flag 
indicating the validity of data is always sent along with the message retrieved from the database. This 
approach to abbreviating the blockchain is a common idea among developers, particularly those building 
proof-of-concept blockchain implementations, but it is vulnerable sińce a malefactor can always shift 
their point of attack from the blockchain to the NoSQL database. 

In order for the application to communicate with the blockchain validator nodes, each application serwer 
runs a local version of the blockchain. This local version is a non-validator node synchronized with the 
blockchain, it pushes new transactions to validator nodes but does not participate in the consensus. This 
design was chosen to allow validator nodes to focus on adding transactions with consensus rather than 
responding to blockchain ąueries, which would otherwise slow the consensus. We adopted an 
asynchronous approach when pushing new transactions to the blockchain, this allows users to view 
transactions before the transaction is validated by all validators. When adding a new transaction, the 
validity of the transaction is checked by the local blockchain, returning true if the local blockchain 
verifies the transaction as valid. The local blockchain node ‘gossips’ with the validators with consensus 


2 http: / /tendermint.com version »■» 

3 https://en. wikipedia.org/wiki/Gossip protocol 

4 https: //www.mongodb.com 
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being ąuickly found. Subseąuently, the new transactions are added to the blockchain, after which the 
user receives confirmation that the transaction is validated (Buchman, 2016). 

5 BLOCKCHAIN-BASED COLLABORATION TOOL EVALUATION 

Collaboration tools are on-line services used by large audiences and are characterized by countless 
database changes, with the addition of other special reąuirements, such as the need for Iow response 
times and confidential data storage. These reąuirements prove to be a challenge to both security and 
scalability. 

Many commercial message-based groupware applications use blockchain (see Dust r >, Status 5 6 , e-Chat 7 , 
and BeeChat 8 9 ), some with the purpose of creating an immutable, distributed permanent record of 
communication and others with the intent of an agreed Peer-to-peer protocols for message deletion. 
Spasovski (with Andreassen and Lyck) developed dallr? to create a simple intuitive platform with a fiat 
learning curve to target smali to medium sized companies. dallr is a groupware communication 
application that provides the platform for our empirical comparison. Two test scenarios are designed to 
simulate large numbers of users communicating via the collaboration tool. Performance and scalability 
are the two key features measured. Performance is evaluated via response-time, scalability under 
increasing load and the serwer resources consumed. The difference between the two test scenarios lies 
in the form or topology of communication between the users. The first uses a realistic messaging pattern 
based on a scale-free network (Barabasi et al., 1999). The second simulates a client-server topology 
where a single node (server/authority/command and control node) receives messages from the entire 
client-network. 

Each test scenario was run for 60 seconds including a 10 second ramp-up period in which all threads 
are started and thereafter run concurrently. The time-out for all reąuests is 20 seconds throughout all 
tests. Response-time and throughput are the preferred metrics to test both ‘SendMessage’ and 
‘RetrieveAllMessages’ functions. SendMessage appends a message to those previously sent. 
RetrieveAllMessages is the analogue of restarting the groupware client and reloading every message ever 
sent between two parties into the groupware application. Both non-blockchain and blockchain 
implementations are tested for each of the network topology scenarios, scale-free and centralised. 

Database Instance 


Remote 

Desktop 


Local Machinę 


Figurę 1: Oueruiew ofthe testing enuironment. 
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5 https://usedust.com 

6 https: / / status.im 

7 https://echat.io 

8 https://beechat.io 

9 https://bitbucket.org/nosai /dallr-scala/src/blockcham-testing-tendermmt-nonblocking/ 
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We create the first testing scenario by applying network theory, specifically scale-free network theory to 
produce scale-free networks. We use the Barabasi-Albert (1999) model to produce power-law graphs 
using a Java implementation of the Barabasi-Albert model named GraphStream 10 . Each user is 
represented as a vertex and a message sent from one user to another is represented as an (undirected) 
edge. A node with relatively high degree of connectivity is denoted a ‘hub’. 

The first test scenario receives the number of users as input and creates that many threads (users) that 
run in parallel. The scale-free network graph is loaded into the test which instructs each user who to 
message using the ‘SendMessage’ function. Each ‘SendMessage’ function has a 10% chance to trigger 
‘RetrieveAllMessage’ function, followed by a 300-millisecond delay. Once the test reaches the end, it 
waits for (#users*2) milliseconds before restarting. 

We now describe the second scenario, namely 1 to n-i (or centralized client-server topology). By forcing 
all users to send their messages to a single user (User #1), we replicate a heavy load onto a single user. 
By comparing the load on the heavily loaded user with the remaining users in both implementations, we 
emphasize the blockchain's ability to cope with centralized stress. The test takes the number of users as 
input and creates that many threads (users) run in parallel. Each user concurrently sends a message to 
User #1, using ‘SendMessage’, followed by 300 milliseconds delay. Thereafter the user has a 10% chance 
to retrieve all messages using the ‘RetrieveAllMessage’ function, followed by a 300 milliseconds delay. 

As seen in Figurę 1, all reąuests are sent using 10 slave test-machines controlled by the master testing 
serwer. Each slave concurrently sends reąuests to an ‘Application Load balancer’ which distributes 
reąuests across application servers. The blockchain implementation runs a local blockchain used to 
communicate and synchronized with the validator blockchains. A single database serwer is shared 
between all application serwers. The tests, using Apache JMeter 11 , ran up to 800 concurrent users for 
over a minutę. Microsoft Azure 12 was used as the cloud infrastructure for all serwers. The 10 blockchain 
validator nodes are deployed in three different geographic regions (Amsterdam, London and Frankfurt) 
on the DigitalOcean cloud 1 -. In this way, we attempt to replicate the real-world, where the blockchain 
nodes would be distributed between a consortium of dispersed organizations. 

Buchman (2016) ran detailed tests on Tendermint showing that by adding morę validators, this both 
lowers the throughput and increases latency. He first ran 64 validators and achieved throughput of 
4,000 transactions per second with latencies of 2 seconds, and then 8 validators with 9,000 transactions 
per second and 1.5 second latency. The number of instances that run the application and local blockchain 
are the only factors that change within the testing infrastructure. The tests are run initially with one 
instance and the load gradually increased until results show average response times of over a second. 
After 1 second average response times appear, the number of instances is doubled and the tests are 
restarted and run until an average response time of ower 1 second re-appears. This process is repeated 
until an average response time of over 1 second appears on 8 test instances. Both implementations are 
tested on 4 cloud configurations with of 1, 2, 4 and 8 instances. 

6 DISCUSSION OF EXPERIMENTAL RESEARCH 

This section presents experimental results in the form of linę graphs focusing on system performance, 
measured in response times and throughput for different topologies and resourcing. Ramsay et al. 
(1988) famously noted negative user behavior with response times greater than 200 milliseconds. We 
focus on maintaining a response time of less than 20oms and do not display results over 350 
milliseconds. We used linę graphs for comparing blockchain and non-blockchain implementations in 
terms of different user loads using the scale-free network and 1 to n-i test scenario. The two messaging 
functions tested in each scenario are the ‘RetrieveAllMessages’ function and the ‘SendMessage’ as 
previously described. 

The metric to measure scalability is the average response time and throughput under an increasing user- 
loads. In our ewaluations, ‘User #1’ represents the heavily loaded user that receives all messages in the 1 
to n-i test scenario while ‘User i’ represents each of the remaining n-i users. The users in this test are 
represented as a rooted tree where each user is a node and each edge a message sent. The user receiving 
all messages is the root node of the rooted tree, all the remaining users are leaf nodes. This allows us to 
compare the average response times of the ‘RetrieveAllMessages’ function being called on the blockchain 


10 http: / / graphstream-proiect.org/ 

11 http: / /imeter.apache.org 

12 https://azure.microsoft.com/en-au/ 

*3 http: //www.digitalocean.com 
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and non-blockchain implementations. This evaluation illustrates how the two implementations handle 
the stress of a single heavily loaded user. 

We anticipated that the function 'RetrieveAl 1 Messages’ would be the bottleneck on the blockchain 
implementation. This is due to the large number of ąueries the blockchain performs within the function. 

The blockchain data is stored using Merkle-tree data structure, andthus it takes 0 (log 2 n) time to search 
a błock containing n transactions. The function 'RetrieveAl 1 Messages’ retrieves all the user’s messages 
(m) from the database, and then ąueries the blockchain. This implies a running time of 0 (m log 2 n) 
added to the database ąuery. The send message function performs a single insertion to the database, as 
well as a single cali to the function TnsertToblockchain’, the latter being a constant time operation. 

Due to the asynchronous naturę of how the implementation adds messages to the blockchain, we 
expected the sending of messages to perform well. Insertion to the database is expected to be much 
slower than ąuerying due to write locks on the database (Nyati et al., 2013). 

We did not expect to return consistent results due to unpredictable network traffic on the Microsoft 
Azure cloud that can cause unexpected response times and inconsistent latencies. Evidence of 
unpredictable network traffic on the Microsoft Azure cloud is visible and latencies throughout tests can 
randomly double over any 45 second period. A graphical illustration of our evaluation and measuring 
results can be seen in Figurę 2 & 3. 


Average Throughput for SendMessage Function 
(1 Instance) 

■ Blockchain (Scalę Free) •— Non-Blockchain (Scalę Free) 


Average Throughput for RetrieveAIIMessagesO Function 

— Blockchain (Scalę Free) ■ • Non Blockchaln (Scalę Free) 

-» Blockchain (1 to N) * Non-Blockchaln (1 to N) 


— Blockchain (1 to N) ■■ Non-Blockchain (1 to N) 




Figurę 2: Comparing the throughput of‘SendMessage’ and ‘RetrieueAllMessages’functions on 
both Scale-Free&i to n-i topologies and implementations (blockchain and non-blockchain) on a 
single D3 (4 Core 13 GB RAM) instance. 


Average Response Time on Scalę Free Network Test Scenario (1 Instance) 


Average Response Time on 1 to N Test Scenario (1 Instance) 



Users 


Users 


Figurę 3: Scale-Free Network Test: compares ‘RetrieueAllMessages’for both Scale-Free & 1 to n- 
1 topologies for both implementations (blockchain and non-blockchain) on a one x D3 (4 Core 13 
GB RAM) instance. 

The results scalę linearly until they reach their threshold, and then have an exponential ratę of growth. 
This is due to the time-outs when the implementation can no longer handle the transaction load at the 
given resource level. 

In most tests, the majority of reliable results are those under 200 milliseconds on average and the 
scalability of both implementations is linear. As a concrete example, a single instance of 
‘RetrieveAllMessages’ has a threshold of 50 users, whereas 8 instances can handle 400 users, namely 
the function scales linearly. 
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We observe that the non-blockchain implementation can handle 4 to 8 times the load of the blockchain 
implementation before thresholding, and this blockchain tool incurs a penalty of 2 to 4 times on 
response time. Results also showed that in the blockchain application the ‘SendMessage’ function keeps 
pace with the non-blockchain tool. This is due to the asynchronous non-blocking design pattern choice 
our tool follows. The ‘SendMessage’ function does not wait until each message is validated and added to 
the blockchain before retuming to work. 


Average Response Time on Scalę Free NetWork Test Scenario 
(8 Instances) 


■ Blockchain RetrieveAIIMessages() * Blockchain SendMessage!) 

— Ar-Non-Blockchain RetrieveAIIMessages() —— Non-Blockchain SendMessage!) 



SO 100 200 400 800 

Users 


Average Response Time on 1 to N Test Scenario (8 Instances) 


■ ■ Blockchain RetfleveAllMessages() ♦ ■ Blockchain SendMessageO 
w Non Blockchain RetrleveAIIMessagesQ —*— Non-Blockchaln SendMessageO 



Users 


Figurę 4: Scale-Free Network Test: compares ‘RetrieueAllMessages’for both Scale-Free &iton- 
1 topologies for both implementations (blockchain and non-blockchain) on eightxD3 (4 Core 13 
GB RAM) instances. 


We also conclude that the ‘RetrieveAllMessages’ function on the blockchain tool has a tougher time 
coping with the same user load on the 1 to n-i test scenario than in the scale-free network topology. Ali 
ąuerying of the blockchain occurs through the local blockchain which cannot handle the load directed 
on a single user in the 1 to n-i test scenario. 


Max Send Throughput -1 to N Test 



Instances 


Max Load Throughput -1 to N Test 



Instances 


Figurę 5 (left) The ‘SendMessage’function on 1 to n-i network (Blockchain &Non-Blockchain) 
showing how the function scales ouer D3 instances 1-8 and (right) the ‘RetrieueAllMessages’ 
function on 1 to n-i network (Blockchain & Non-Blockchain) showing how the function scales 
ouer D3 instances 1-8. 

The non-blockchain application has linear throughput growth throughout all tests while the blockchain 
application grows linearly until a threshold is reached. The throughput is throttled on both functions of 
the blockchain tool due to ‘RetrieveAllMessages’ function timing out. The testing software is built to run 
on threads which disallow the test to continue until the time out is finished. The response time of 
‘RetrieveAllMessages’ starts increasing dramatically from 100 users which is where the linear growth of 
the throughput stops. In Figurę 2 (right) it is visible that a blockchain root node has a polynomial ratę 
of growth while the non-blockchain root node seems to be constant. The leaf nodes are almost identical 
regardless of the implementation. This proves that blockchains are capable of keeping up with non- 
blockchain Solutions when they are not under heavy load. 
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Max Send Throughput - Scalę Free Test Max Load Throughput - Scalę Free NetWork Test 



Instances 


Figurę 6 (left) The ‘SendMessage’function Scale-Free network (Blockchain & Non-Blockchain) 
on a single D3 (4 Core 13GB RAM) instance (left) and (right) the ‘RetrieueAllMessages’function 
on a Scale-Free network (Blockchain & Non-Blockchain) on a single D3 (4 Core 13GB RAM) 
instance. 


7 CONCLUSION 

The paper shows that the blockchain implementation of a groupware communication application scales 
linearly as the number of instances is increased, until it reaches the throughput threshold. We also 
learned that using the Tendermint blockchain incurs a 4-8 multiplicative factor on scalability when used 
with our cloud-based groupware application, and a multiplicative factor of 2-4 on the average response 
times and throughput in the application. 

These two findings imply that high throughput and Iow response times can still be achieved if enough 
servers are deployed to resource the groupware communication application. For well-established 
companies concerned with transaction immutability and ledger security, paying 4 to 8 times morę in 
cloud computing costs is unlikely to be a concern. 

We have also shown that blockchains perform poorly when handling heavy traffic loads through single 
centralized users. This implies - naturally enough - that centralized topologies are morę vulnerable to 
cyber-attacks, such as denial of service, because they expose a single point of failure, which when placed 
under stress, has an amplifying effect in the remainder of the network. Indeed, such a network traffic 
model is better suited to a trusted 3 rd party server model rather than a distributed ledger 
implementation. 

This paper reflects on the testing method for a single cloud-based blockchain-enabled application for 
groupware communication. The testing parameters are the number of applications, the ąuantity of 
messages sent and the topology of the message traffic. The resources consumed are in terms of servers 
and processes deployed, the transaction throughput and the network latency. The ambition of our futurę 
research is to provide a generał methodology for ąuantifying and anticipating cloud-based resources and 
overall system performance when deploying blockchain-enabled applications. A further ambition is to 
use this experience to compare the performance when using different blockchain frameworks in 
applications. This in order to better inform developer choices. 
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